Three-Dimensional Overlays Within Navigable Panoramic Images, and Applications Thereof

ABSTRACT

A panorama viewer is disclosed which facilitates navigation from within the panorama of a larger, structured system such as a map. The panorama viewer presents a viewport on a portion of a panoramic image, the viewport including a three-dimensional overlay rendered with the panoramic image. As the orientation of the viewport within the panoramic image changes, the three-dimensional overlay&#39;s orientation in three-dimensional space also changes as it is rendered with the panoramic image in a manner that matches the change in orientation of the viewport.

This application is a continuation of U.S. patent application Ser. No.11/754,267, filed May 25, 2007, which is incorporated by referenceherein in its entirety.

FIELD OF THE INVENTION

The present invention generally relates to displaying imagery, and moreparticularly to viewing and navigating within panoramic images, andapplications thereof.

BACKGROUND

Computerized mapping systems traditionally provide a top-downpresentation of the mapping data. Enhancing the mapping system withstreet-level imagery presents various interface challenges: such asdealing with navigation within the street-level view, including turningat intersections, and correlating the user's location and orientation ona map with the user's location and orientation within the street-levelview, both in absolute terms (e.g., latitude, longitude, and heading)and relative to landmarks like city streets and intersections. Recently,A9 BlockView (no longer online) and Windows Live Local TechnologyPreview powered by Microsoft Virtual Earth(http://preview.local.live.com) have attempted to provide a usableinterface to street-level views of a city. A9 BlockView addressed theorientation problem by flattening imagery into two strips, so that theuser does not have the freedom to look around 360 degrees. Windows LiveLocal presents a “car” view of street-level imagery which is rotated in90 degree increments by manipulating a car avatar on the map view.

BRIEF SUMMARY OF THE INVENTION

The present invention relates to viewing and navigating within panoramicimages, and applications thereof. In an embodiment, a panorama viewer isdisclosed which facilitates navigation from within the panorama of alarger, structured system such as a map. The panorama viewer presents aviewport on a portion of a panoramic image, the viewport including athree-dimensional overlay rendered with the panoramic image. As theorientation of the viewport within the panoramic image changes, thethree-dimensional overlay's orientation in three-dimensional space alsochanges as it is rendered with the panoramic image in a manner thatmatches the change in orientation of the viewport. For example, wherethe panoramic image is of street-level imagery, the three-dimensionaloverlay can comprise a three-dimensional representation of annotationsto a map, including three-dimensional representation of the lines thatrepresent the streets on the map.

Further embodiments, features, and advantages of the present invention,as well as the structure and operation of the various embodiments of thepresent invention, are described in detail below with reference to theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS/FIGURES

Embodiments of the invention are described with reference to theaccompanying drawings. In the drawings, like reference numbers mayindicate identical or functionally similar elements.

FIG. 1 is a diagram of an exemplary distributed system suitable forpracticing an embodiment.

FIG. 2 is a diagram illustrating an example of how a mapping service canbe integrated with a panorama viewer, in accordance with an embodiment.

FIG. 3 depicts an example of a browser display, in accordance with anembodiment.

FIG. 4 is a flowchart illustrating exemplary processing performed by apanorama viewer, in accordance with an embodiment.

FIG. 5 depicts exemplary Extensible Markup Language (XML) configurationinformation.

FIG. 6 illustrates an example of a panoramic image.

FIGS. 7A, 7B, and 7C illustrate user interaction with the panoramaviewer viewport.

FIG. 8 is a flowchart of processing performed by a renderer, inaccordance with an embodiment.

FIGS. 9A, 9B, and 9C illustrate a relationship between a surface, aprecomputed region, and a viewport.

FIGS. 10A and 10B illustrate a simple example of generatingtransformation parameters.

FIG. 11 depicts a panorama which has been warped, in accordance with anembodiment of the invention.

FIG. 12 depicts an exemplary transformation based on yaw and pitch forforming the panorama of FIG. 11.

FIG. 13 depicts an exemplary panorama image displayed in accordance withan embodiment of the invention.

FIG. 14 is a diagram illustrating how to generate coordinates for a userannotation, in accordance with an embodiment.

FIG. 15 is a flowchart of processing performed in the generation of userannotation coordinates, in accordance with an embodiment.

The present invention is described with reference to the accompanyingdrawings. The drawing in which an element first appears is typicallyindicated by the leftmost digit or digits in the corresponding referencenumber.

DETAILED DESCRIPTION

The present invention relates to viewing and navigating within panoramicimages, and applications thereof. In the detailed description of theinvention herein, references to “one embodiment”, “an embodiment”, “anexample embodiment”, etc., indicate that the embodiment described mayinclude a particular feature, structure, or characteristic, but everyembodiment may not necessarily include the particular feature,structure, or characteristic. Moreover, such phrases are not necessarilyreferring to the same embodiment. Further, when a particular feature,structure, or characteristic is described in connection with anembodiment, it is submitted that it is within the knowledge of oneskilled in the art to effect such feature, structure, or characteristicin connection with other embodiments whether or not explicitlydescribed.

FIG. 1 is a distributed system suitable for practice of an embodiment ofthe invention. A client 110 communicates with one or more servers 150,for example, across a network such as the Internet or a local areanetwork. Client 110 can be a general-purpose computer with a processor,local memory, a display, and one or more input devices such as akeyboard or a mouse. Alternatively, client 110 can be a specializedcomputing device such as, for example, a mobile handset. Server(s) 150,similarly, can be implemented using any general-purpose computer capableof serving data to client 110.

Client 110 executes a panorama viewer 120, the operation of which isfurther described herein.

As illustrated by FIG. 1, panorama viewer 120 requests configurationinformation 130 from server(s) 150. As discussed in further detailherein, the configuration information includes meta-information about apanorama to be loaded, including information on links within thepanorama to other panoramas. In an embodiment, the configurationinformation is presented in a form such as the Extensible MarkupLanguage (XML). Panorama viewer 120 retrieves visual assets 140 for thepanorama, for example, in the form of panoramic images or in the form ofpanoramic image tiles. In another embodiment, the visual assets includethe configuration information in the relevant file format. Panoramaviewer 120 presents a visual representation on the client display of thepanorama and additional user interface elements, as generated fromconfiguration information 130 and visual assets 140, as furtherdescribed herein. As a user interacts with an input device to manipulatethe visual representation of the panorama, panorama viewer 120 updatesthe visual representation and proceeds to download additionalconfiguration information and visual assets as needed.

In an embodiment, panorama viewer 120 can be a standalone application,or it can be executed within a browser 115, such as Mozilla Firefox orMicrosoft Internet Explorer. Panorama viewer 120, for example, can beexecuted as a script within browser 115, as a plug-in within browser115, or as a program which executes within a browser plug-in, such asthe Adobe (Macromedia) Flash plug-in. In an embodiment, panorama viewer120 is integrated with a mapping service, such as the one described inU.S. Pat. No. 7,158,878, “DIGITAL MAPPING SYSTEM”, which is incorporatedby reference in its entirety herein.

FIG. 2 illustrates an example of how a mapping service 210 can beintegrated with panorama viewer 120. Mapping service 210 displays avisual representation of a map, e.g., as a viewport into a grid of maptiles. Mapping system 210 is implemented using a combination of markupand scripting elements, e.g., using HTML and Javascript. As the viewportis moved, mapping service 210 requests additional map tiles 220 fromserver(s) 150, assuming the requested map tiles have not already beencached in local cache memory. Notably, the server(s) which serve maptiles 220 can be the same or different server(s) from the server(s)which serve panorama tiles 140 or the other data involved herein.

In an embodiment, mapping service 210 can request that browser 115proceed to download a program 250 for panorama viewer 120 from server(s)150 and to instantiate any plug-in necessary to run program 250. Program250 may be a Flash file or some other form of executable content.Panorama viewer 120 executes and operates as described above.Alternatively, configuration information 130 and even panorama tiles 140can be retrieved by mapping service 210 and passed to panorama viewer120. Panorama viewer 120 and mapping service 210 communicate so as tocoordinate the operation of the user interface elements, to allow theuser to interact with either panorama viewer 120 or mapping service 210,and to have the Change in location or orientation reflected in both.

FIG. 3 is an example browser display 300 that presents both a mappingservice such as mapping service 210 and an integrated panorama viewersuch as panorama viewer 120. The mapping service provides a button 310entitled “Street View” that, when selected, preferably changes theappearance of the map in areas where panorama data is available. Forexample, in FIG. 3, streets with available panorama data arehighlighted. This highlighting can be, for example, a colored and/orshaded outline or overlay, or a change in color and/or shading. This canbe implemented by using a transparency image with the map tile or bydirectly including the effect in the map tile served to the mappingservice.

The mapping service allows a user to activate the panorama viewer byfurther selecting a point on the map. When a point is selected by theuser, a character or avatar icon 320 is displayed at the point on themap. In an embodiment, the avatar icon includes an indicator of whatdirection the avatar icon is facing, which in FIG. 3 is represented asan arrow underneath the avatar icon.

In an embodiment, as the panorama viewer is instantiated by the mappingservice, the panorama viewer is presented in the form of viewport 330embedded in an informational balloon window associated with avatar icon320. The orientation of the visual representation of the panorama withinviewport 330 matches the orientation of avatar icon 320. As the usermanipulates the visual representation of the panorama within viewport330, the panorama viewer informs the mapping service of any changes inorientation or location so that the mapping service can update theorientation and location of avatar icon 320. Likewise, as the usermanipulates the orientation or location of avatar icon 320 within themapping service, the mapping service informs the panorama viewer so thatthe panorama viewer can update its visual representation.

In an embodiment, the viewport 330 of the panorama viewer presents apanoramic image of the selected area. The user can click and drag aroundon the image to look around 360 degrees. In the example viewport 330depicted in FIG. 3, a variety of user interface elements are added tothe underlying panorama. These elements include navigation inputs suchas, for example, zoom and panning controls (e.g., navigation buttons) onthe left side of the viewport and annotations in the form of lines/bars,arrows, and text that are provided directly in the panorama itself. Theannotations are rendered in a three dimensional manner that roughlymatches the three dimensional scene depicted in the panorama.

In FIG. 3, for example, the lines/bars in viewport 330 correspond to thestreets depicted in the corresponding map and can even be rendered inthe same color as the streets depicted in the map. The arrows areselectable by a user (by clicking or by dragging along the street line),one going in each direction that there is another panorama available.These allow the user to navigate up and down the street (i.e., to changethe vantage point from which the street is viewed). As the user looksaround 360 degrees, the lines and arrows smoothly track the underlyingimagery so that the lines remain on top of the underlying streets, andso that the arrows are always visible on the screen. This allows theuser to navigate along the street while looking straight ahead, or whilelooking to the side of the storefront.

When the user clicks on an arrow to navigate within the viewport, azooming cross-fade effect and other visual cues give the user a sense ofmovement. When the user arrives at an intersection of two streets, thereis one green line and two arrows for each street. All of these arevisible at the same time, and all are labeled, so that the user knowsthe current location and can proceed in any direction. This techniquecan readily scale to accommodate complex intersections with more thanfour directions. When the user reaches a “dead end” where the roadcontinues but no further imagery is available, there is only one arrowon the street indicating the direction in which the user can navigate.In the other direction, a symbol and message embedded in the image canbe presented to inform the user that imagery is not available in thisdirection.

The user interface is not restricted to navigating along a line to walkdown a street and can be readily extended to allow users to stray fromthe line elements when useful: for example, to cross over to theopposite side of the street to get a closer look at something. Moreover,there are environments within a city where a user might be expected todesire to snap off of a street and navigate freely within an adjacentarea, for example, a park, plaza, shopping area, or otherpedestrian-friendly public place. The interface can be readily enhancedwith “free movement zones” to provide this functionality. It should alsobe noted that although the user interface is presented in the context ofnavigation between discrete street-level panoramas, it could equallywell be used to allow a user to navigate through a more continuous setof panoramic data, such that navigating along a street would be assmooth as video.

The operation and implementation of the user interface elements aredescribed in further detail below.

FIG. 4 is an exemplary flowchart of processing performed by a panoramaviewer such as, for example, panorama viewer 120, in accordance with anembodiment of the invention.

At step 402, the panorama viewer receives an identifier for the initialpanorama to be presented and various viewport parameters, such as thesize of the viewport and the orientation to be viewed within thepanorama. This information can be passed to the panorama viewer from amapping service, e.g., by using Flashvars or ExternalInterface betweenthe Flash plug-in and the Javascript in the mapping service.

At step 404, the panorama viewer uses the panorama identifier to requestconfiguration information from the server (e.g., an XML file). FIG. 5depicts exemplary XML configuration information 500 and is discussed infurther detail below. The XML is parsed, and the information is loadedinto various data structures for use by the panorama viewer. In anembodiment, the XML includes information for the panorama viewer such asdata properties and projection properties of the current panorama, andinformation on annotations/links within the panorama, including links toother panoramas.

At step 406, the panorama viewer requests the visual assets for thepanorama and stores the received visual assets, for example, in localmemory/storage. In an embodiment, the panorama viewer can maintain acache of visual assets and limit bandwidth usage to retrieval of visualassets which are not in the cache. FIG. 6 illustrates an example of aninitial panoramic image 600. The complete panoramic image 600 can beretrieved by the panorama viewer, or panoramic image 600 can be dividedinto multiple panorama image tiles and the tiles requested only asneeded by the panorama viewer.

At step 408, the panorama viewer processes the configuration informationand the visual assets to prepare for rendering the visual representationof the panorama in the viewport at step 410. With regard to the visualassets, the panorama viewer can assemble the panorama image tiles intothe portion of the complete panoramic image which overlaps with theviewport. The panorama viewer can present the panoramic image as a flatsurface or as a texture-mapped three dimensional surface such as, forexample, a cylinder or a sphere, as further discussed herein. Withregard to the annotations overlay presented in the viewport, thepanorama viewer uses the configuration information to compute the shapesand locations for these various elements such as, for example, thelines/bars and the arrows presented in the viewport.

In an embodiment, the polygons/shapes are modeled in a three-dimensionalspace that corresponds to the space depicted in the panorama. Thesepolygons/shapes can be modeled, for example, using a pinhole cameramodel (e.g., the focal length can be generated by multiplying the heightof the viewport by a constant relative depth of the center of rotation).The polygons/shapes of the annotations overlay change their orientationin the three-dimensional space in a manner that matches the change inorientation of the viewport. In one embodiment, the polygons/shapes arerotated by an angle equal to the difference between the currentorientation of the user's point-of-view in the panorama and thedirection of the annotation, as specified in the configurationinformation. The polygons/shapes can be further transformed arounddifferent spatial axes in order to take into account non-flat panoramas,as further described herein.

At step 410, the visual representation of the panorama in the viewportis rendered.

At step 412, the panorama viewer receives and manages input, forexample, by capturing input events such as mouse and keyboard events.The panorama viewer, for example, detects whether the user has pannedthe viewport (e.g., by dragging the mouse or by selecting a pan controlbutton), has selected to zoom (e.g., by clicking on the panorama or bymoving the zoom slider control on the left of the viewport with themouse) or has selected a link to another panorama (e.g., by clicking onan arrow with the mouse).

At step 420, a determination is made regarding whether a user has pannedthe viewport. If the user has panned the viewport, control transfers tostep 422. If the user has not panned the viewport, control transfers tostep 430.

At step 422, the panorama viewer determines whether the viewport willoverlap with any panorama image tiles which will need to be retrievedfrom either the server or a cache.

At step 424, the panorama viewer executes the necessary computations forallowing the viewport correctly to be rendered in a differentorientation, as further described in detail herein.

At step 426, the panorama viewer notifies the mapping service of the neworientation selected by the user so that the mapping service can updateits avatar icon's facing indicator. The panorama viewer re-computes theshapes and locations for the viewport elements and renders the viewport.To illustrate this point, consider FIG. 7A which depicts the panoramaviewer viewport from FIG. 3. FIG. 7B shows the result after a user hasselected to pan the panorama to the left. Note that the lines/bars thatcorrespond to the roads depicted in the panorama change theirorientation as the panorama viewer changes the orientation of thepanorama.

At step 430, a determination is made regarding whether a user has zoomedthe viewport. If the user has zoomed the viewport, control transfers tostep 432. If the user has not zoomed the viewport, control transfers tostep 440.

At step 432, the panorama viewer determines, for example, whether torequest new higher resolution panorama image tiles from a server (orfrom cache), or whether to utilize existing tiles at a differentclose-up resolution, for example, where no such higher resolution tilesexist.

At step 434, the viewport parameters are changed to reflect thedifferent zoom level. A transition can be provided between the zoomlevels so as to give the appearance of actively zooming into the nextzoom level of the panorama. FIG. 7C shows the result after the user hasselected to zoom in on a feature in FIG. 7A.

At step 440, a determination is made regarding whether a user hasselected a link to another panorama. If the user has selected a link toanother panorama, control transfers to step 442. If the user has notselected a link to another panorama, control transfers to step 412.

At step 442, the panorama viewer proceeds to begin the process oftransitioning between the original panorama and the new panorama. Thepanorama viewer can, for example, zoom the original panorama and performa cross-fade to the new panorama to give the user a sense of movement.Alternatively, the panorama viewer can play an actual video transitionbetween the two panoramas.

At step 444, the panorama viewer notifies the mapping service of the newlocation selected by the user so that the mapping service can update itsavatar icon's location and can scroll the map accordingly.

In embodiments, the panorama viewer can be implemented using anyadvantageous programming language or style of programming. For example,the panorama viewer can be implemented using object-oriented programmingwith separate classes designated to handle the XML configurationinformation, the annotations, the texture generation, the tilemanagement, and the mesh generation.

FIG. 5 sets forth exemplary XML configuration information 500 (e.g.,metadata). As illustrated by the example shown in FIG. 5, the schema forthe configuration information is organized into “data_properties”,“projection_properties”, and “annotation_properties”.

The subgroup Data_Properties contains attributes such as “pano_id”(e.g., a unique identifier for the panorama), “image_width” and“image_height” (e.g., dimensions of the panoramic image before beingsplit into tiles), “tile_width” and “tile_height” (e.g., dimensions ofthe tiles), “lat” and “lng” (e.g., coordinates of the current panorama),and “num_zoom_levels” (e.g., the number of zoom levels that the userwill be able to view in the panorama viewer). This subgroup alsocontains elements such as “text” (e.g., that can be used to representthe street name of the current panorama), “copyright” (e.g., copyrightinformation), and “street_range” (e.g., the range of numbers in thegiven street).

The subgroup Projection_properties contains attributes such as“Pano_yaw_deg” (e.g., orientation of the vehicle which captured theimages which generated the panoramic image), “tilt_yaw_deg” and“tilt_pitch_deg” (e.g., the yaw and pitch of the line of highest slopewhich, as further described herein, is useful for dealing with panoramaswith sloped features), and “vertical_scale” (e.g., fraction of the imagealong the y-axis that is visible at the lower zoom level).

The subgroup Annotation_properties contains attributes such as“horizon_height_fraction” (e.g., vertical position (height) of thehorizon, expressed as a fraction of the visible strip, which can beadjusted to maximize the fit between the annotations and the imagery ofthe tiles) and “annotation_height_fraction” (e.g., vertical position(height) of the plan containing the annotations, expressed as a fractionof the visible strip). This subgroup also includes the “pano_link”subgroup which describes properties of link symbols that allow a user tonavigate to a neighboring panorama or to another related document. The“link” subgroup includes “link_text” (e.g., description of the landingpanorama) as an element and includes the following as attributes:“yaw_deg” (e.g., direction that the link is pointing to), “pano_id”(e.g., identifier to linked panorama), and “road_argb” (e.g., anattribute of the road, such as the color of the road on the map) (notshown). The subgroup can also include a “floating_text” group or elementwhich identifies arbitrary features in the panorama and could alsoprovide for an arbitrary link, for example to a local data repository ora website (not shown).

It should be noted that the above schema for the configurationinformation is merely illustrative and can be arranged in any of anumber of advantageous ways, including using techniques that do not relyon XML.

FIG. 8 is a flowchart of processing performed by a renderer inaccordance with an embodiment of the invention.

At step 802, the renderer precomputes a pre-image of a viewport by abackward transform. This defines a portion of a surface, which isreferred to herein as the “precomputed region”. FIG. 9A illustrates thisin the context of a cylindrical surface 900 with a rectangular viewport920, Which defines a precomputed region 910 by the backwardtransformation. It should be noted that the viewport does not have to berectangular and that the technique works for discretized cylinders(based on a mesh) or for continuous voxel-to-pixel mappings. Forexample, a mesh can be defined on the viewport with a corresponding meshon the cylinder. These meshes do not have to be uniform, and are imagesof one another as defined by the forward or backward mappings. The meshon the cylinder will typically only cover a portion of the cylinder. Inthe case of a continuous transformation, the pre-image of the viewportwould define a continuous region of the cylinder.

At step 804, the renderer precomputes the transform, which maps eachpixel from the precomputed region to a pixel in the viewport. In asense, it is assumed that the cylinder is standing still in space.Instead of attaching a texture image to a changing cylinder, it isassumed that the texture will “slide” on the cylinder.

At step 806, the renderer translates an image/texture in response to auser input.

At step 808, the renderer determines that portion of the image/texturethat intersects the precomputed region of the surface. This defines theset of pixels that need to be rendered in the viewport. If the user haschanged the point of view recently, then this needs to be updated. Moreprecisely, any panning to the left or right of the viewport can bereadily achieved by translating the texture in the correspondingdirection, at step 806, thereby generating a different intersection withthe precomputed region. Likewise, any panning up or down is achieved bytranslating the texture along these directions. Any arbitrary directionof panning is achieved by simply translating the texture in thecorresponding direction. Each time, a new intersection with theprecomputed region is generated. This is illustrated in FIGS. 9B and 9C,where 950 represents the precomputed region and 960 represents theimage/texture.

At step 810, the precomputed transform is applied to the portion ofimage/texture that intersects with the precomputed region.

Finally, at step 812, the transformed imagery is rendered into theviewport.

In an embodiment, the renderer utilizes properties of rectilinearprojections to speed up the rendering of the panoramic images. If asurface like a cylinder is viewed as infinite, then it is a groupendowed G with the natural operations (e.g., translation along axis, androtation around axis). Likewise, the texture, if viewed as infinite, isalso a group H endowed with translations in the plane. It turns out thatthere is a canonical homomorphism between G and H. In other words, arotation of the cylinder around its axis is equivalent to as translationof the texture, for example, in the x-direction. A translation of thecylinder along its axis is equivalent to a translation of the texture,for example, in the y-direction. This allows one to pre-compute allprojection parameters in advance and to simulate a change of viewpointas a translation of the texture. FIG. 10A and 10B illustrate an exampleof how to compute projection parameters from the screen space to thetexture space. As illustrated by FIGS. 10A and 10B:

-   -   (1) If a point M on screen 1020 has coordinates (x, y), then in        space it has coordinates (x, y, R), where R is the radius of the        cylinder 1010.    -   (2) In this case,

${{\tan \; \theta} = \frac{x}{R}},$

and the point P has, in the texture space, the following coordinates:

$P = {( \frac{\arctan ( \frac{x}{R} )}{\frac{Ry}{\sqrt{x^{2} + R^{2}}}} ).}$

A dynamic texture based on the current zoom level can be generated andpositioned in the image space. This texture changes when the userchanges the point-of-view (e.g., by zooming or panning). This texturecan be obtained by concatenating tiles from a tile pyramid at theappropriate level and scale. If some tiles are missing, one can fallback on a tile at a parent level in the tile pyramid. The texture ismodeled as being mapped over a cylinder. A projection is performed overthe screen space. This nonlinear projection can be approximated as apiecewise affine transformation. More precisely, the cylinder and screenspaces can be discretized using a triangular mesh. Each triangle can berendered by linearly (or rather, affinely) mapping a piece of thetexture over it. This is well-defined as an affine transform in the twodimensional plane since it is uniquely determined by its action on threepoints (hence, the use of triangles). The mesh can be made uniform inthe screen space (and not in the texture space). The screen mesh isalways the same regardless of the zoom level. Different texture meshescan be used depending on the zoom level. For each triangle, a texturemesh corresponds to a unique triangle in the screen mesh and a unique(affine) transformation matrix. Such a matrix can be pre-computed as theproduct of a screen matrix and (the inverse of) a texture matrix.

When a user pans, all the renderer needs to do is adjust and/or refreshthe texture. This is fast, because it consists of memory copies. Copyinglarge chunks of pixels is usually highly optimized in severalprogramming languages.

In an embodiment, zooming in consists of dividing both horizontal andvertical fields of vision by two and using the next zoom level togenerate the texture. When the user zooms in/put, the panorama viewercan pre-cache a few bitmap images to make the animation smooth. As faras the projection itself, one can use the various sets of transformationmatrices at integral zoom levels. At non-integral zoom levels, one canlinearly interpolate the transformation matrices (still fast) betweenthe previous zoom level and the next one.

In a case where pixels are assumed to be square, they correspond tohomogeneous solid angles. The elementary field of vision of a givenpixel is the same in the horizontal and the vertical directions. Thisenables a trade-off to be made. For example, one can choose exactly topreserve straight lines, but this will result in higher distortion onthe side. Alternatively, one can decide to have straight lines become abit curvy, and thereby reduce the amount of distortion. In anembodiment, the height of the original image is scaled to that of theviewport. Because the pixels are square, the ratio of the width overheight of the viewport determines the horizontal field of vision.

In the case of a sphere rather than a cylinder, the assumptions aboveare no longer true. Accordingly, the above technique alone cannotsimulate a true pan up/down, because all that is performed is a movealong the cylinder's axis with the vision vector perpendicular to thisaxis. Nevertheless, the motion of a true pan up/down can be simulated bypre-computing a series of transforms and linearly interpolating betweenthe transforms.

In an embodiment, the panorama viewer is configured to handle non-flatpanoramas. Not all panoramas depict flat and horizontal features, e.g.,consider many of the streets of San Francisco. Cameras mounted on avehicle, for example, used to capture panoramas are parallel to theground. Thus, traveling on a steep incline can result in misorientedpictures. Accordingly, in such situations, it can be advantageous towarp the panorama so as to ensure that vertical buildings in the realworld remain vertical in the texture space. FIG. 11 depicts an exampleof how a panorama 1100 can be so warped. As illustrated by FIG. 12, theexample roughly follows a periodic function which can be used to guidethe placement of the viewport as well as the generation of theannotations in a manner that takes the slope of the panorama intoaccount.

As noted, herein, the configuration information can include projectionproperties such as the yaw and the pitch of the highest slope in thepanorama. As illustrated by FIG. 12, the panorama viewer can use the yawand pitch of the direction of the steepest slope to constrain theviewport to the sinusoidal strip of the warped panorama. The renderingof the annotation elements in the viewport also can be modified to takeinto account the yaw and pitch information of the slope of the panorama.The spatial orientation of the lines/bars and arrows can be transformedbased on the yaw and pitch information or, alternatively, can beestimated based on the relative yaw of the annotation and the yaw of thesteepest slope. FIG. 13 illustrates the result of such processing ofconfiguration information on the slope of a panorama. The panoramacorrectly places the vertical buildings in the panoramic image on thesteeply-sloped street. Moreover, the line/bar (e.g., street linemetadata) depicting the road is tilted at an angle which roughly matchesthe slope of the street.

In an embodiment, the panorama viewer also is able to facilitate userannotations to a panorama image. User annotations to panoramas representa challenge with respect to how to reference an annotation inthree-dimensional space.

FIGS. 14 and 15 illustrate an embodiment which addresses userannotations in three-dimensional space. The processing illustrated inFIG. 15 can occur at the panorama viewer (or the mapping service), atthe server, or a combination of the two.

Referring to FIG. 15, at step 1502, a user inputs an annotation withrespect to one panorama. The panorama viewer can receive the user inputin any of a number of different ways, including by receiving a clickevent on the spot on the panorama which the user desires to annotate.The two-dimensional location of the annotation on the panorama isrecorded in some advantageous coordinate system, e.g., by location onthe panorama image or by yaw and pitch coordinates.

At step 1504, the user navigates to another nearby panorama in thepanorama viewer, locates the same feature to be annotated, and againinputs an annotation with respect to the second panorama. The panoramaviewer or the mapping service can presumably also offer the ability toadd additional metadata associated with the annotation, such as a title,link, graphics, etc.

At step 1506, the annotations coordinates on the two panoramas are usedto generate three-dimensional coordinates for the annotation. Given theknown position of the cameras which took the images for the panorama1410, 1420 and the user-input annotation coordinates relative to thetwo-dimensional images, as illustrated in FIG. 14, it is possible tocompute the intersection of the two, depicted as 1450. The result is athree-dimensional coordinate for the annotation.

At step 1508, the three-dimensional coordinate for the annotation isassigned to the annotation and stored in the database of annotations.The annotation can then be included with any panorama within someadvantageous range of the computed coordinates, including panoramaswhich were not originally annotated by the user.

Alternatively, where the relative pitch information is not particularlyimportant to an annotation, it is possible to receive the userannotations as a one-dimensional yaw direction on both panoramas, whichfacilitates the assignment of a two-dimensional geocode to theannotation (with or without default pitch information).

While various embodiments of the present invention have been describedabove, it should be understood that they have been presented by way ofexample, and not limitation. It will be apparent to persons skilled inthe relevant art(s) that various changes can be made therein withoutdeparting from the scope of the invention. Furthermore, it should beappreciated that the detailed description of the present inventionprovided herein, and not the summary and abstract sections, is intendedto be used to interpret the claims. The summary and abstract sectionsmay set forth one or more but not all exemplary embodiments of thepresent invention as contemplated by the inventors.

The foregoing description of the specific embodiments will so fullyreveal the general nature of the invention that others can, by applyingknowledge within the skill of the art, readily modify and/or adapt forvarious applications such specific embodiments, without undueexperimentation, without departing from the general concept of thepresent invention. Therefore, such adaptations and modifications areintended to be within the meaning and range of equivalents of thedisclosed embodiments, based on the teaching and guidance presentedherein. It is to be understood that the phraseology or terminologyherein is for the purpose of description and not of limitation, suchthat the terminology or phraseology of the present specification is tobe interpreted by the skilled artisan in light of the teachings andguidance.

1. A method for viewing panoramic images, comprising: (a) receiving atleast a portion of a first panoramic image geocoded to a first location;(b) receiving configuration information specifying an overlay for thefirst panoramic image; (c) rendering, in a three-dimensional space, theat least a portion of the first panoramic image for display; (d)rendering the overlay in the three-dimensional space of the firstpanoramic image for display with the first panoramic image in aviewport; (e) in response to receipt of a first input to changeorientation of the viewport, changing the orientation of the overlay inthree-dimensional space as it is rendered with the first panoramic imageso as to match a change in orientation of the first panoramic imagewithin the viewport; and (f) in response to receipt of a second inputspecifying a position on the first panoramic image, presenting a secondpanoramic image geocoded to a second location different from the firstlocation, the second location determined according to the position. 2.The method of claim 1, wherein the rendering (d) comprises modeling theoverlay using a pinhole camera model, wherein a focal length of thepinhole camera model corresponds to a height of the viewport and a depthof the three-dimensional space.
 3. The method of claim 1, wherein therendering (c) comprises texture mapping the at least a portion of thefirst panoramic image to a surface having canonical homomorphism.
 4. Themethod of claim 1, wherein the overlay comprises an annotationdescribing a feature of the first panoramic image, the annotationpositioned to correspond to a position of the feature in the firstpanoramic image.
 5. The method of claim 4, wherein the annotation is astreet line positioned to correspond to the direction of a street in thefirst panoramic image, and wherein the overlay further comprises astreet name positioned to identify the street in the first panoramicimage.
 6. The method of claim 1, wherein the overlay includes anavigation icon associated with the second panoramic image and whereinthe second input in (f) is a selection of the navigation icon.
 7. Themethod of claim 1, wherein the receiving (a) comprises receivingstreet-level imagery.
 8. The method of claim 1, further comprising: (g)displaying a user orientation character at a first position on a map,the first position corresponding to the first location; and (h) inresponse to the second input, displaying the user orientation characterat a second position on the map, the second position corresponding tothe second location.
 9. The method of claim 8, further comprising: (j)presenting a first line on the map representing a first street wherepanoramic image data is unavailable; and (k) presenting a second line onthe map differently from the presenting in (j), the second linerepresenting a second street where panoramic image data is available,whereby the difference in the presenting (j) and the presenting (k)indicates to a user where panoramic image data is available.
 10. Asystem for viewing panoramic images, comprising: a panorama viewer that(i) receives at least a portion of a first panoramic image andconfiguration information, the first panoramic image geocoded to a firstlocation, and the configuration information specifying an overlay forthe first panoramic image, (ii) renders, in a three-dimensional space,the at least a portion of the first panoramic image and the overlay ofthe first panoramic image for display in a viewport, (iii) in responseto receipt of a first input to change orientation of the viewport,changes the orientation of the overlay in three-dimensional space as itis rendered with the first panoramic image so as to match a change inorientation of the first panoramic image within the viewport, and (iv)in response to receipt of a second input specifying a position on thefirst panoramic image, presents a second panoramic image geocoded to asecond location different from the first location, the second locationdetermined according to the position.
 11. The system of claim 10,wherein the panorama viewer models the overlay using a pinhole cameramodel to render the overlay in the three-dimensional space, wherein afocal length of the pinhole camera model corresponds to a height of theviewport and a depth of the three-dimensional space.
 12. The system ofclaim 10, wherein the panorama viewer models texture maps the at least aportion of the first panoramic image to a surface having canonicalhomomorphism to render the first panoramic image.
 13. The system ofclaim 10, wherein the overlay comprises an annotation describing afeature of the first panoramic image, the annotation positioned in thefirst panoramic image to correspond to a position of the feature. 14.The system of claim 13, wherein the annotation is a street linepositioned to correspond to the direction of a street in the firstpanoramic image, and wherein the overlay further comprises a street namepositioned to identify the street in the first panoramic image.
 15. Thesystem of claim 10, wherein the overlay includes a navigation iconassociated with the second panoramic image and wherein the second inputis a selection of the navigation icon.
 16. The system of claim 10,wherein the first and second panoramic images comprise street-levelimagery.
 17. The system of claim 10, further comprising: a mappingservice that displays a user orientation character at a first positionon a map, the first position corresponding to the first location and, inresponse to the second input, displays the user orientation character ata second position on the map, the second position corresponding to thesecond location.
 18. The system of claim 17, wherein the mapping servicepresents a first line on the map representing a first street wherepanoramic image data is unavailable and presents a second line on themap differently from the presentation of the first line, the second linerepresenting a second street where panoramic image data is available,whereby the difference in the presentation of the first line and thepresentation of the second line indicates to a user where panoramicimage data is available.
 19. A system for serving panoramic images andmetadata, comprising: a server that sends data to a client, the dataincluding at least a portion of a panoramic image geocoded to alocation, a configuration information specifying an overlay for thepanoramic image, and a file, wherein the file instructs the client torender, in a three-dimensional space, the at least a portion of thepanoramic image for display and to render the overlay in thethree-dimensional space of the panoramic image for display with thepanoramic image in a viewport, wherein the file instructs the client, inresponse to receipt of a first input to change orientation of theviewport, to change the orientation of the overlay in three-dimensionalspace as it is rendered with the panoramic image so as to match a changein orientation of the panoramic image within the viewport, and whereinthe file instructs the client, in response to receipt of a second inputspecifying a position on the panoramic image, to present a secondpanoramic image geocoded to a second location different from the firstlocation, the second location determined according to the position. 20.The system of claim 19, wherein the file is a Flash file.